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A pilot research study searched for appropriate 
methods to evaluate instructional television (ITV) , The specific 
objectives were to find ways to measure: 1) the degree to which the 
media presentation's objectives were being met^ and 2) the appeal of 
the show, as judged by viewers* attention. Five intermediate grade 
level educable mentally retarded (EHH) students viewed three segments 
of the "Electrc Company", Three methods attempted to measure the 
degree to which the subjects learned what was being taught — a 
multiple choice posttest, an individually administered posttest 
presented via flash cards, and direct questions asked during the 
presentation; observers recorded the viewers' frequency of attention 
and described their behavior. The achievement results were 
inconclusive, due to small sample size and the sophistication of the 
viewers vis-a-vis the show, although the multiple choice test 
appeared the least appropriate technique. Both attention measures 
were useful, A replication of the study using primary level educable 
mentally retarded subjects and normal primary students would be 
useful, (Author/PB) 
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EVALUATION OF ITV: ELECTRIC COMPANY PILOT 
Margery A, Mac Donald 

ABSTRACT 



The purpose of this pilot study vas to find appropriate methods 
for the evaluation of instructional television. The subjects > five 
intermediate level educable mentally retarded students > were shown 
three video-taped segments of the "Electric Company/' Measures of 
both attention and achievement were utilized in three trials. The 
results, while inconclusive due to the sma}I sample size, indicate 
that the most effective achievement measures were the individually 
administered test and the stop-tape interview. Both attention measures 
used — recording of frequency of subjects watching and written obser- 
vations of behavior — were found to be good indicators of appeal; 
however, 'ihe observational data yielded more insight into the inter- 
action between subject and media. 



SPECIAL REPORT No. 7210 
COMPUTER-BASED PROJECT for the EVALUATION 
of MEDIA for the HANDICAPPED 

T i t I 8 I EVALUATION OF ITV: ELECTRIC COMPANY PILOT 
BY: Margery A. MacDonald 

BACKGR OUND 

The Computer Based Project for the Evaluation of Media for the Handicapped, 
based on contract <iOEC-9-423017-4357 (616) between the Syracuse (N.Y.) City School 
District and tJie Media Services and Captioned Films Branch, Bureau of Education 
for the Handicapped (United States Office of Education) for the five year period 
July 1, 1969 through June 30, 1974 • The major goal is to improve the instruction 
of handicapped children through the development and use of an evaluation system 
to measure the instructional effectiveness of films and other materials with 
educable mentally handicapped (EMH) children, in-service training and media support 
for special teachers, and studios related to the evaluation process and the 
populations used. 

The Project has concentrated on the 600 films and 200 filmstrips from the 
Media Services and Captioned Films (BEH - USOE) depository; however, specific 
packages from Project LIFE, various elementary math curricula, and selected 
programs from Children's TV Workshop have also been evaluated* The evaluation 
model used requires that: 1) objectives of materials be specified and written; 

2) instruments be constructed to test and measure effectiveness; and; 3) children 
be the major sources of evaluation information, A number of instruments and 
methodologies are employed in the gathering of cognitive and affective data from 
900 EMH children and 80 special teachers to make the effectiveness decisions. 
Over half oi the EMH population can neither read or write; therefore, a unique 
Student Response System (SFxS) iy employed, consisting of a twenty station G.E.- 
1000 SRS which can be operated in a group or individual recording mode and is 
connected to a remote computer system. The computer capabilities consist of 
remote telephone connections to the Rome IN.YJ Air Development Command, the 
Honeywell time-shared network, and the Schenectady (N.Y.) G E Research and 
Development Center; and batch mode capabilities of tho Syracuse City Schools, 
Syracuse University, and various commercial sources. 

In-servico and media support activities provide on-the-job training for 
teachers, teacher aides, equipment, and materials to the special teachers in 
the city schools. The research activities have centered around investigations 
and Special problems related to the development of the evaluation model. The 
four major areas considered are: 1) testing effects, 2) captioning effects, 

3) special student characteristics; and, 4) evaluation procedures validation. 

Documentation of the major activities appear in the five annual reports 
and the 600 evaluations prepared on materials used. Staff members v;ere encouraged 
to prepare special reports and the attached paper is one of these. The opinions 
expressed in this publication do not necessarily reflect the position or policy 
of the Computer Based Project, the United States Office of Education, or the 
Syracuse City School District, and no official endorsement by any of the agencies 
should be inferred. 
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EVALUATION OF ITV: ELECTRIC COMPANY PILOT 

The purpose of this pilot evaluation was to find an appropriate 
method for evaluating Instructional television. Therefore, more atten- 
tion should be given to the measures employed In this study than to 
the results. 

In evaluating any form of media, it Is necessary to deter- 
mine the extent to which educational goals and objectives are being met. 
More often than not^ the objectives of a piece of media are not clearly 
outlined prior to production and must be determined from viewing the 
finished product. In the case of the Electric Company , however, the 
Children's Television Workshop (CTW) research staff has done a comrriend- 
able job of stating, in behavioral terms, the specific objectives of 
the series. Part of the evaluation problem, then, was to determine the 
extent to which these stated objectives were being achieved. 

The second Important consideration in the evaluation of instruct- 
ional television Is appeal. Is the show sufficiently attractive to gain 
the attention of its audience? It would seem logical that the closer 
children attend to a show, the more they wi}l learn from It. Therefore, 
the pilot evaluation of Electr ic Company included measures of both atten 
tion and achievement. 



METHOD 



5 Intermediate EMH students whose reading scores on the 
Wide Range Achievement Test (WRAT) ranged from 2.0 - 2»6* 
The same subjects were used for all three trials. 
Three one-half hour show segments were video taped for 
use In the study (shows #5^, /?56, and #57) • 

Objective questions were photographed and presented In a 
slide presentation format. Tf^e data on these questions 
v/as gathered via the G, E. Student Response System. (See 
Append ix A. ) 

Flashcards were prepared for the Individual achievement 
test. 

DESIGN; Three trials were run, each using a different combination 

of achievement and attention measure. 
TRIAL A set of multiple-choice questions were used 

as both a pre-test and a post-test* The criteria for 
achievement of an objective was a 20% pre/post gain and 
60% correct on an answer (post-test only). Attending 
behavior was measured by having an observer record, at 10 
second Intervals, the number of children watching the media 
TR I AL /2 : Achievement was measured by stopping the tape 
at predetermined points and asking a ch 11 (whose name was 



SUBJECT: 



EQmPHENT: 



selected prior to the trial) what he was learning. The 
child's response was scored on a scale from I to 3 and an 
observation of the group response was made, (See Appendix 
B) . Attention was recorded tn the same way as In Trial ^1. 

TRIAL Achievement was measured using an Individually 

administered test presented on flash cards. The pre-test 
was given three days before the showing, the post-test 
Immediately after* Attention was measured by recording 
observations made of the children during each show segment. 

RESULTS 

TRIAL ft\ {Show ffS^) Gain score analysis for this trial 
v/as considered Inappropriate as the pre-test showed prior 
knowledge of %0% or greater on eight of the thirteen I terns • 

Attention was rated 76 (out of a possible lOO) for 
students watching* 

TRIAL #2: (Show #56) The stop-tape interview revealed 
that most of the students in the sample understood what 
was being taught by the media The weakest show segment 
was the "punctuation*^ sequence. (See Appendix B.) 

Attention was rated 90 (out of a possible lOO) for 
students watching. 

TRIAL #3i (Show #57) Results of the Individual achieve- 
ment tests are reported in Table /-I. 



The attention observation Is Included herein as 
Appendix C. Overall attention to this show was high, 
with several children participating verbally throughout* 

PJ^SCUSSION 

The results of all achievement measures were Inconclusive. One 
explanation may be that the subjects used In this pilot were too soph- 
isticated for the material being presented. S's were chosen on the 
basis of reading scores on the WRAT, administered over a year ago. 
Results of a more recent WRAT Indicate that thereadtng level of this 
group ranges from 2,^ to k.6y which may explain the high level of prior 
knowledge evidenced on all achievement measures. 

Among the achievement measures » the experimenter adjudged the 
Individual test and the stop-tape interview to be the most effective, 
and the mul tipfe-cholce question format least effective in measuring 
the objectives. The flash card technique, unlike the multiple-choice 
question^ allows a child to demonstrate a number of reading skills, 
e, g, making consonant blends The stop-tape technique not only allows 
the child an opportunity to verbalize what he Is learning, but a\so 
seems to encourage attention to the media. In trial #2, the subjects 
were alerted to the fact that the tape v^ould be stopped and seemed to be 
enthusiastically awaiting t!^e;r turn to respond throughout the showing, 
For these reasons, the e><perlneriter did not consider this technique 
disruptive but, rather, beneficial to the learning process. 



Both attention data gathering techniques were found to be effect- 
ive. However, while '*eyes -on-screen** data often measures close attcnti 
to the media, It cannot give Information on children's verbal and motor 
reactions to show segments. These kinds of responses, e.g» reading a 
sign aloud» are noteworthy, especially when evaluating a reading show, 
and provide valuable Inputs Into the evaluation process. 

RECOMMENDATIONS 

The experimenter considers the findings of this pilot to be of 
limited Vc^lue due to; 1) Inappropriate selection of sample, and 2) 
small sample size. In order to further Investigate methods of evalua^- 
tlng Instructional television, the experimenter recommends that the 
pilot be repeated on two other samples; 1) a normal first grade group, 
and 2) a primary EMH group. 
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TRIAL //3 Pre and Post-test scores 

on Ind Ivldual test . 



TABLE I 
ELECTRIC COMPANY PILOT 



Question Student Student ill Student #3 Student ffh Student ffS 
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CONCEPT 

Short ' r' sound 



APPENDIX A 

ELECTRIC COMPANY PILOT EVALUATION 

OBSERVATION #1 
Achievement Measure 
Show »Sh 



BEHAVIORAL OBJECTIVE 
The child can demonstrate his knowledge of 
Individual letter/sound correspondence by 
blending the sounds In simple linear sequence 
to produce intelligible words. He can do 
this following a simple blending model or a word 
fami ly model . 



QUESTIONS 

I) Which word sounds like bit? 

1 . car 

2. hat 

3. cab 
hit 



2) The "i" In did sounds like the "i" in: 

1. bite 

2. light 
^'3. grip 

A. site 



CONCEPT 



BEHAVIORAL OBJECTIVE 



ch^ sound 



The child can recognize certain groups of 
letters as single units and process them as 
such when sounding out words* 



QUESTIONS 




3) 1. curtain 

2, clothespfn 
'^'^i • chicken 



h) Tester says: ^Whlch word begins with the sound "ch^' 

1, cloth 

2, crack 
'<3. church 

crank 



CONCEPT 



''all family 



QUESTIONS 



5) B+ all 



BEHAVIORAL OBJECTIVE 
The child can recognize larger spelling 
patterns as single units and process them 
as such when sounding out words. 



1. <r>j2=^bat) 

2, {C) (ball) 



CONCEPT 



BEHAVIORAL OBJECTIVE 



Sight word: The child can recognize certain groups of 

letters as simple units and process them as 
such when sounding but words, 

QUESTION 

6) Tester says: "Which of the following Is the word 'Ms'*? 

K if 

2. It 

>'^3. Is 

^. In 



CONCEPT BEH AVIORAL OBJECTIVE 

Morpheme - ing The child can Interpret some high frequency 

Standard English morphemes when presented 1 
appropriate context, 

7) Choose the correct answer: 

1 » The boy was sat . 
2, The boy was sltted, 
i'<3* The boy was si tt ing, 

8) Choose the correct answer : 



^>2. 
3. 



The top is spin. 
The top Is spinning^ 
The top spirt. 



CONCEPT 



BEHAVIORAL OBJECTIVE 



Sight word: jf The child can recognise certain groups of 

letters as single units and process them as 
such when sounding out words, 

QU ESTION 

9) Taster says: '%/hich of the following Is the word If** 

1, Is 

2. in 
>V3, If 

It 



CONCEPT 



BEHAVIORAL OBJECTIVE 



Silent 



QUESTIONS 

10) Tester says 

I • wane 

2, win 

3* wing 

^i*. wine 



The child recognizes the following structural 
spelling patters and can successfully read 
words containing them: Ffnal **e" signal Ifng 
a 'Mong'* vowel sound. 

^Whlch of the following is the word wlne?^' 



n) Tester says: ^ Which of the following Is the word 'Wde?^' 



made 
2, mad 



CONCEPT 



BEHAVIORAL OBJECTIVE 



Capital letters and 



The child can use the following punctuation 



periods as cues 



cues in interpreting sentences: a sentence 



ends with a 



a ? or an 



\ . 



a sentence 



begins with a capital letter. 



QUESTIONS 



12) Pick the correct sentence: 



1 . The bal 1 • lost the boy 

2. The boy the bal I . lost 
>'0. The boy lost the bal I . 



13) Pick the correct sentences 



I. The ocean. swtns In the fish 
>^2, The fish swims In the ocean. 
3. Swins In the ocean, the fish 
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APPENDIX C 



E LECTRIC COMPANY PILOT 
Trial n 



OBSERVATION OF GROUP BEHAVIOR 
Show ,^57 
(N=5) 



Observations were made by show segment (28 total). All verbal 
responses were recorded In quotation marks. In most Instances, attri- 
bution to the speaker was recorded also. 

During "Electric Company' theme the children were very attentive. 
Most of them mouthed or sung the words . One child (Abraham) read aloud 
the sign "Show 57 • 



SEGMENTS 
Shopping bags 

Water 

Restaurant Scene 
With 

Snake (th) 
Path 

Faces (th blend) 



REACTIONS 

"Wag'' All attentive. Several ..louth the words. 
"Wit'' 'W - I - t, wet." "Wit, wit, wit, witi" 

Two children read aloud, "Water." 

"Walter." All attentive. "Walter" "I want 
water" "Well." (Abraham) "Walter wants 
water" (Abraham laughs.) 



All read "with." Attentive. "W 
Lllllanne.) 



h" 



Jerry looks around. Abraham, "th" "ma- th - 
math," "path," "bath." Three of the children 
smi 1e at the ending. 

"Path." 

Vivian and Abraham make al I blends aloud. Lil 1 1 
anne mouths them. -Russel attentive but non- , 
verbal. Jerry inattentive. 



Thud 



"Thud." 



Think (etc.) 
Man In Box 

Haunted House 



That doesn't swing 
Not Safe for Swimming 

I am talking 
ALK block 

Marashlsh I 



Hey 1 Walk careful ly 
Walk - Don't Walk 

I will notl 



He would be a 
nice pet 



'Think. Thick. Thud.'^ Vivian and Jerry 
Inattentive. 

V/e've seen this.'' (Vivian) Abraham Inatten- 
tive. What's that say?^' (Jerry.) 'Looks 
different. Several kids looking around room. 
Three of the kids laugh at the ending. 

Three kids read It aloud. No response to ending. 

^'Not Safe to Swim. ''Not safe for swimming.'' 
(Abraham and Vivian.) Attention good. Vivian 
smi les at ending. 

'I am talking. (Abraham, Lllllanne, Vivian) 
'm talking.'' (Abraham) 

*'0oh| ooh| we've seen this before. (Abraham) 
^^Ooh, a}k .... talk, walk, chalk, ' (All of 
the grou^^.) 

Three of the kids attending. Abraham leaning 
back In chair, eyes closed. Begins paying 
attention by middle of segment. All children 
say "four alks aloud. Vivian looks out window. 
Abraham says "stuck'' for stalk. Vivian sucks 
her thumb . 

Al 1 eyes on screen , 

"Walk.^^ Don^t Walk.' Read by several kids 
before read aloud on show. 

I won't. You will.'' (Abraham) Abraham laughs. 
Shouts with characters on screen. LilUonne 
laughs . Vivian smi 1 es . 

Several kids mouth the words , Attention falls 
off quickly. 



I can*t swim 



''I can't '^ '^1 can't. 'Can't swim. Vivian 
and Abraham inattentive. 



S 4 



The Early Bfrd Catches 



Worm 
Supper 

Vi 's Diner 

Rock Bank 
Wi )d Guess 

Love of Chair 

The Last Word 



Abraham pieces It together at ffrst sight, 
"The Early Bird catches the worm, A period 
spozed to go at the end/- (Vivian) Vivian 
looks out the window, Jerry looks at his 
feet. The others are Inattentive, Vivian 
sucks thumb, Abraham Inattentive, Abraham, 
^ That makes sense there, ^ The early bird 
cathces the worm, (Abraham) Repeats this 
over and over. All the kids are fidgeting, 
rocking In their seats, slumping^ swinging 
thel r legs etc 

"Worm^* (everyone) 

''Supper, supper' (All) ''Supper time 
(Abraham) Super . (Abraham) 

*'Super, Super, Abraham and Jerry are fooling 
around, Vivian lying back In seat All the 
kids are fidgeting. Several yawn, Vivian 
looks at the celling Jerry yawns, Vivian 
sucks her thumb. All become more attentive 
toward the end, 

Vivian nods her head In time to the music, 
"Dinner, Olner. Lllllanne laughs, Abraham 
smiles and repeats the words, Jerry Is atten- 
tive, 

Lllllanne yawns, Jerry and Vivian look around. 
Others are attentive but do not react, Abraham 
becomes Inattentive. Jerry Is listening then 
fidgets. Lilllanne tells Jerry to sit still, 
Jerry and Abraham are fidgeting. Wild guess. ' 

'As our story begins *. (Vivian) Vivian and 
Abraham read and say most of the sentences, 
Attention very high. What about Haoml? 
(Vivian) "Love of Chair, (Jerry) Russel 
Hits Vivian, 

• Wave^ (All) Vivian waves . 



